Assessing Robustness of a Deep Learning Model for COVID-19 Classification on Chest Radiographs

The COVID-19 pandemic has made a dramatic impact on human life, medical systems, and financial resources. Due to the disease's pervasive nature, many different and interdisciplinary fields of research pivoted to study the disease. For example, deep learning (DL) techniques were employed early to assess patient diagnosis and prognosis from chest radiographs (CXRs) and computed tomography (CT) scans. While the use of artificial intelligence (AI) in the medical sector has displayed promising results, DL may suffer from lack of reproducibility and generalizability. In this study, the robustness of a pre-trained DL model utilizing the DenseNet-121 architecture was evaluated by using a larger collection of CXRs from the same institution that provided the original model with its test and training datasets. The current test set contained a larger span of dates, incorporated different strains of the virus, and included different immunization statuses. Considering differences in these factors, model performance between the original and current test sets was evaluated using area under the receiver operating characteristic curve (ROC AUC) [95% CI]. Statistical comparisons were performed using the Delong, Kolmogorov-Smirnov, and Wilcoxon rank-sum tests. Uniform manifold approximation and projection (UMAP) was used to help visualize whether underlying causes were responsible for differences in performance between test sets. In the task of classifying between COVID-positive and COVID-negative patients, the DL model achieved an AUC of 0.67 [0.65, 0.70], compared with the original performance of 0.76 [0.73, 0.79]. The results of this study suggest that underlying biases or overfitting may hinder performance when generalizing the model. © 2023 SPIE.

Decision region analysis to deconstruct the subgroup influence on AI/ML predictions

Assessing the generalizability of deep learning algorithms based on the size and diversity of the training data is not trivial. This study uses the mapping of samples in the image data space to the decision regions in the prediction space to understand how different subgroups in the data impact the neural network learning process and affect model generalizability. Using vicinal distribution-based linear interpolation, a plane of the decision region space spanned by the random 'triplet' of three images can be constructed. Analyzing these decision regions for many random triplets can provide insight into the relationships between distinct subgroups. In this study, a contrastive self-supervised approach is used to develop a 'base' classification model trained on a large chest x-ray (CXR) dataset. The base model is fine-tuned on COVID-19 CXR data to predict image acquisition technology (computed radiography (CR) or digital radiography (DX) and patient sex (male (M) or female (F)). Decision region analysis shows that the model's image acquisition technology decision space is dominated by CR, regardless of the acquisition technology for the base images. Similarly, the Female class dominates the decision space. This study shows that decision region analysis has the potential to provide insights into subgroup diversity, sources of imbalances in the data, and model generalizability. © 2023 SPIE.

Police stress: the role of occupational culture and officer characteristics

PurposeThe purpose of the current study was to augment the police culture and stress literature by empirically examining the impact of features of the internal and external work environment, as well as officer characteristics, on police officer stress.Design/methodology/approachThe current empirical inquiry utilized survey data collected from street-level officers in a mid-sized urban police department in a southern region of the United States (n = 349).FindingsThis study revealed that perceived danger, suspicion of citizens and cynicism toward the public increased police occupational stress, while support from supervisors mitigated it. In addition, Black and Latinx officers reported significantly less stress than their White counterparts.Research limitations/implicationsWhile this study demonstrates that patrol officers' perceptions of the external and internal work environments (and race/ethnicity) matter in terms of occupational stress, it is not without limitations. One limitation related to the generalizability of the findings, as results are gleaned from a single large agency serving a metropolitan jurisdiction in the Southeast. Second, this study focused on cultural attitudes and stress, although exact connections to behaviors are more speculative. Finally, the survey took place prior to the onset of the COVID-19 pandemic and the killing of George Floyd (and others), which radically shook police–community relationships nationwide.Practical implicationsPolice administrators should be cognizant of the importance that views of them have for patrol officer stress levels. Moreover, police trainers and supervisors concerned with occupational stress of their subordinates should work toward altering assignments and socialization patterns so that officers are exposed to a variety of patrol areas, in avoiding prolonged assignments of high social distress.Originality/valueThe study augmented the police culture and stress literature by empirically uncovering the individual-level sources of patrol officers' job-related stress. This study builds off of Paoline and Gau's (2018) research using data collected some 15 years ago by examining a more contemporary, post–Ferguson, context.

Design Concerns for Multiorganizational, Multistakeholder Collaboration: A Study in the Healthcare Industry: JMIS

Multiorganizational, multistakeholder (MO-MS) collaborations that may span organizational and national boundaries, present design challenges beyond those of smaller-scale collaborations. This study opens an exploratory research stream to discover and document design concerns for MO-MS collaboration systems beyond those of the single-task collaborations that have been the primary focus of collaboration engineering research. We chose the healthcare industry as the first target for this research because it has attributes common to many MO-MS domains, and because it faces significant challenges on a global scale, like the recent COVID-19 pandemic, for which MO-MS collaboration could offer solutions, as, for example, evidenced by the rapid collaborative development and distribution of COVID-19 vaccines. To this end, we reviewed 6,609 articles to find 100 articles that offered insights about the design of MO-MS collaboration systems, then conducted 50 semi-structured interviews in two countries with expert practitioners in the field. From those sources, we derived an eleven-category set of design concerns for MO-MS collaboration systems and argue their generalizability to other MO-MS domains. We offer exemplar probe questions that designers can use to increase the breadth and depth of requirements gathering for MO-MS collaboration systems.

Establishment and application of ERA-LFD method for rapid detection of feline calicivirus.

Feline calicivirus (FCV) has a single-stranded, positive-sense RNA genome, and it is responsible for many infectious respiratory diseases in cats. In addition, more worryingly, highly virulent strains of FCV can cause high mortality in felines. Therefore, a rapid and reliable diagnosis tool plays an important role in controlling the outbreak of FCV. In this study, enzymatic recombinase amplification (ERA) assay combined with lateral flow dipstick (LFD) was developed for the detection of FCV, targeting a relatively conversed position of FCV-ORF1. The results showed that the optimal reaction condition was at 40 °C for 30 min. ERA-LFD method was highly sensitive with the detection limit as low as 3.2 TCID50 of FCV RNA per reaction. The specificity analysis demonstrated no cross-reactivity with feline parvovirus (FPV), feline herpesvirus (FHV) and feline infectious peritonitis virus (FIPV). ERA-LFD was highly repeatable and reproducible, with the intra-assay and inter-assay coefficients of variation for this method both less than 7%. The general test showed that all the recombinant plasmids with known mutant sites and FCV strains with different mutant sites stored in our laboratory were all detected by this method. Of the 23 samples, 14 samples were tested positive for FCV by ERA-LFD and RT-qPCR, respectively. In summary, ERA-LFD assay was a fast, accurate and convenient diagnosis tool for the detection of FCV. KEY POINTS: • The detection principle of ERA-LFD was introduced. • Almost all the currently known FCV strains can be detected. • ERA-LFD is easy to operate and can be used for field detection.

How Generalizable and Interpretable are Speech-Based COVID-19 Detection Systems?: A Comparative Analysis and New System Proposal

Recent work has shown the potential of using speech signals for remote detection of coronavirus disease 2019 (COVID-19). Due to the limited amount of available data, however, existing systems have been typically evaluated within the same dataset. Hence, it is not clear whether systems can be generalized to unseen speech signals and if they indeed capture COVID-19 acoustic biomarkers or only dataset-specific nuances. In this paper, we start by evaluating the robustness of systems proposed in the literature, including two based on hand-crafted features and two on deep neural network architectures. In particular, these systems are tested across two international COVID-19 detection challenge datasets (COMPARE and DICOVA2). Experiments show that the performance of the explored systems degraded to chance levels when tested on unseen data, especially those based on deep neural networks. To increase the generalizability of existing systems, we propose a new set of acoustic biomarkers based on speech modulation spectrograms. The new biomarkers, when used to train a simple linear classifier, showed substantial improvements in cross-dataset testing performance. Further interpretation of the biomarkers provides a better understanding of the acoustic properties of COVID-19 speech. The generalizability and interpretability of the selected biomarkers allow for the development of a more reliable and lower-cost COVID-19 detection system.

Comparing mental and physical health of U.S. veterans by VA healthcare use: implications for generalizability of research in the VA electronic health records.

OBJECTIVE: The Department of Veterans Affairs' (VA) electronic health records (EHR) offer a rich source of big data to study medical and health care questions, but patient eligibility and preferences may limit generalizability of findings. We therefore examined the representativeness of VA veterans by comparing veterans using VA healthcare services to those who do not. METHODS: We analyzed data on 3051 veteran participants age ≥ 18 years in the 2019 National Health Interview Survey. Weighted logistic regression was used to model participant characteristics, health conditions, pain, and self-reported health by past year VA healthcare use and generate predicted marginal prevalences, which were used to calculate Cohen's d of group differences in absolute risk by past-year VA healthcare use. RESULTS: Among veterans, 30.4% had past-year VA healthcare use. Veterans with lower income and members of racial/ethnic minority groups were more likely to report past-year VA healthcare use. Health conditions overrepresented in past-year VA healthcare users included chronic medical conditions (80.6% vs. 69.4%, d = 0.36), pain (78.9% vs. 65.9%; d = 0.35), mental distress (11.6% vs. 5.9%; d = 0.47), anxiety (10.8% vs. 4.1%; d = 0.67), and fair/poor self-reported health (27.9% vs. 18.0%; d = 0.40). CONCLUSIONS: Heterogeneity in veteran sociodemographic and health characteristics was observed by past-year VA healthcare use. Researchers working with VA EHR data should consider how the patient selection process may relate to the exposures and outcomes under study. Statistical reweighting may be needed to generalize risk estimates from the VA EHR data to the overall veteran population.

From surviving to co-creating: the effects of the reshaped physical and social servicescape on customer citizenship behavior

The pandemic has reshaped customer perceptions of the new normal with both the physical and social service environments. Surprisingly, however, how reshaped servicescape design affects customers, especially their value co-creation behaviors, has not been studied. Drawing on value co-creation and signaling theory, this research aims to examine the comprehensive effects of the physical servicescape (signages, partitions, and spatial density) and the social servicescape (other customer misbehavior) on customer citizenship behavior and revisit intention via the mediating roles of perceived competence, perceived ethicality, and other customer trust. This study conducts two between-subjects experimental design studies with both written and pictorial manipulations in restaurant and retail store contexts to increase generalizability for services marketing. Signages and other customer misbehavior promote customer citizenship behavior through perceived competence and ethicality while partition shows the mixed results on customer perceptions. This paper contributes to servicescape and customer citizenship literature by identifying how the servicescape affects customer citizenship behavior via customers' perception. The findings of this current study also offer practical guidance as to how firms can be more strategic in design choices.

Prognostic models for COVID-19 needed updating to warrant transportability over time and space.

BACKGROUND: Supporting decisions for patients who present to the emergency department (ED) with COVID-19 requires accurate prognostication. We aimed to evaluate prognostic models for predicting outcomes in hospitalized patients with COVID-19, in different locations and across time. METHODS: We included patients who presented to the ED with suspected COVID-19 and were admitted to 12 hospitals in the New York City (NYC) area and 4 large Dutch hospitals. We used second-wave patients who presented between September and December 2020 (2137 and 3252 in NYC and the Netherlands, respectively) to evaluate models that were developed on first-wave patients who presented between March and August 2020 (12,163 and 5831). We evaluated two prognostic models for in-hospital death: The Northwell COVID-19 Survival (NOCOS) model was developed on NYC data and the COVID Outcome Prediction in the Emergency Department (COPE) model was developed on Dutch data. These models were validated on subsequent second-wave data at the same site (temporal validation) and at the other site (geographic validation). We assessed model performance by the Area Under the receiver operating characteristic Curve (AUC), by the E-statistic, and by net benefit. RESULTS: Twenty-eight-day mortality was considerably higher in the NYC first-wave data (21.0%), compared to the second-wave (10.1%) and the Dutch data (first wave 10.8%; second wave 10.0%). COPE discriminated well at temporal validation (AUC 0.82), with excellent calibration (E-statistic 0.8%). At geographic validation, discrimination was satisfactory (AUC 0.78), but with moderate over-prediction of mortality risk, particularly in higher-risk patients (E-statistic 2.9%). While discrimination was adequate when NOCOS was tested on second-wave NYC data (AUC 0.77), NOCOS systematically overestimated the mortality risk (E-statistic 5.1%). Discrimination in the Dutch data was good (AUC 0.81), but with over-prediction of risk, particularly in lower-risk patients (E-statistic 4.0%). Recalibration of COPE and NOCOS led to limited net benefit improvement in Dutch data, but to substantial net benefit improvement in NYC data. CONCLUSIONS: NOCOS performed moderately worse than COPE, probably reflecting unique aspects of the early pandemic in NYC. Frequent updating of prognostic models is likely to be required for transportability over time and space during a dynamic pandemic.

Validation of the Mental Health Changes Indicators Scale against Rasch measurement theory

The Mental Health Changes Indicators Scale (MHCIS) is a brief instrument designed to assess changes in an individual's mental health that occur in response to a specific life event. Although adequate psychometric properties have been demonstrated for this instrument based on classical test theory, the MHCIS has not yet been validated against Rasch measurement theory (RMT). We applied RMT to examine the psychometric properties of the MHCIS using data from 807 Chinese university students. The nominated life event in this study was the COVID-19 pandemic. Despite modest violations of unidimensionality and misfit to the Rasch model, in general, the results supported the validity of the 10-item MHCIS. We recommend further exploration of the generalizability of these results in other populations and across a range of potentially adverse life events.

BACHELOR'S THESIS RATING AS AN ADMISSION CRITERION TO THE MASTER'S PROGRAM OF PSYCHOLOGY: A PSYCHOMETRICAL CASE STUDY

The COVID-19 pandemic influenced admissions testing for the master's degree program of psychology at Masaryk University in 2020. The administration of the standard paper-and-pencil knowledge test was not possible;therefore, we chose the bachelor's thesis ratings instead. This paper is a psychometrical case study that covers the development of criteria related to content validity, design selection, and results. Two randomly selected raters evaluated each thesis, and we equated their severity using a linear logistic test model (LLTM) under the item response theory (IRT) paradigm. This procedure resulted in unidimensional and unbiased scores equated across 18 judges and 2 terms (n1 = 82, n2 = 48). The reliability was comparable to the standard tests, rxx'= 0.869, and judge severity and criteria difficulty did not differ across them. The resulting ratings seem to be valid and no less fair than the written exam. The proposed method can serve other departments and other goals, not only as an entrance test. We share an analytical script and all the necessary materials to enhance the use of this method. © 2022 Masaryk University, Faculty of Arts. All rights reserved.

A systematic review of whether COVID-19 randomized controlled trials reported on demographic and clinical characteristics.

PURPOSE: We aim to assess the reporting of key patient-level demographic and clinical characteristics among COVID-19 related randomized controlled trials (RCTs). METHODS: We queried English-language articles from PubMed, Web of Science, clinicaltrials.gov, and the CDC library of gray literature databases using keywords of "coronavirus," "covid," "clinical trial" and "randomized controlled trial" from January 2020 to June 2021. From the search, we conducted an initial review to rule-out duplicate entries, identify those that met inclusion criteria (i.e., had results), and exclude those that did not meet the definition of an RCT. Lastly, we abstracted the demographic and clinical characteristics reported on within each RCT. RESULTS: From the initial 43 627 manuscripts, our final eligible manuscripts consisted of 149 RCTs described in 137 articles. Most of the RCTs (113/149) studied potential treatments, while fewer studied vaccines (29), prophylaxis strategies (5), and interventions to prevent transmission among those infected (2). Study populations ranged from 10 to 38 206 participants (median = 100, IQR: 60-300). All 149 RCTs reported on age, 147 on sex, 50 on race, and 110 on the prevalence of at least one comorbidity. No RCTs reported on income, urban versus rural residence, or other indicators of socioeconomic status (SES). CONCLUSIONS: Limited reporting on race and other markers of SES make it difficult to draw conclusions about specific external target populations without making strong assumptions that treatment effects are homogenous. These findings highlight the need for more robust reporting on the clinical and demographic profiles of patients enrolled in COVID-19 related RCTs.

How to Churn Deep Contextual Models?

This paper searches for optimal ways of employing deep contextual models to solve practical natural language processing tasks. It addresses the diversity in the problem space by utilizing a variety of techniques that are based on the deep contextual BERT (Bidirectional Encoder Representation from Transformer) model. A collection of datasets on COVID-19 social media misinformation is used to capture the challenge in the misinformation detection task that arises from small labeled data, noisy labels, out-of-distribution (OOD) data, fine-grained & nuanced categories, and heavily-skewed class distribution. To address this diversity, both domain-agnostic (DA) and domain-specific (DS) BERT pretrained models (PTMs) for transfer learning are examined via two methods, i.e., fine-tuning (FT) and extracted feature-based (FB) learning. The FB is implemented using two approaches: non-hierarchical (features extracted from a single hidden layer) and hierarchical (features extracted from a subset of hidden layers are first aggregated, then passed to a neural network for further extraction). Results obtained from an extensive set of experiments show that FB is more effective than FT and that hierarchical FB is more generalizable. However, on the OOD data, the deep contextual models are less generalizable. It identifies the condition under which DS PTM is beneficial. Finally, bigger models may only add an incremental benefit and sometimes degrade the performance. © 2021 ACM.

Generalizability assessment of COVID-19 3D CT data for deep learning-based disease detection.

BACKGROUND: Artificial intelligence technologies in classification/detection of COVID-19 positive cases suffer from generalizability. Moreover, accessing and preparing another large dataset is not always feasible and time-consuming. Several studies have combined smaller COVID-19 CT datasets into "supersets" to maximize the number of training samples. This study aims to assess generalizability by splitting datasets into different portions based on 3D CT images using deep learning. METHOD: Two large datasets, including 1110 3D CT images, were split into five segments of 20% each. Each dataset's first 20% segment was separated as a holdout test set. 3D-CNN training was performed with the remaining 80% from each dataset. Two small external datasets were also used to independently evaluate the trained models. RESULTS: The total combination of 80% of each dataset has an accuracy of 91% on Iranmehr and 83% on Moscow holdout test datasets. Results indicated that 80% of the primary datasets are adequate for fully training a model. The additional fine-tuning using 40% of a secondary dataset helps the model generalize to a third, unseen dataset. The highest accuracy achieved through transfer learning was 85% on LDCT dataset and 83% on Iranmehr holdout test sets when retrained on 80% of Iranmehr dataset. CONCLUSION: While the total combination of both datasets produced the best results, different combinations and transfer learning still produced generalizable results. Adopting the proposed methodology may help to obtain satisfactory results in the case of limited external datasets.

The effect of COVID-19 prevention methods training given through distance learning on state anxiety level: the case of private sector

Purpose>The aim of this study was to determine the effect of coronavirus disease 2019 (COVID-19) prevention methods training given by distance learning technique on the state anxiety level of the workers of a company serving in the communication sector.Design/methodology/approach>The sample of this experimental and cross-sectional study consisted of 52 people working in the communication sector. Data were collected using a questionnaire and the state anxiety inventory. Data were analyzed using descriptive statistics, t-test, variance analysis, Kruskal–Wallis, Mann–Whitney U and Tukey’s test.Findings>While the state anxiety scores of the workers working in the communication sector were 47.94 ± 4.90 before the training, they were found to be 43.98 ± 5.20 after the training.Research limitations/implications>As in every study, this study has some limitations. Although a homogeneous sample is tried to be formed since it only covers this group, it should be considered that there is a limitation in terms of generalizability. In addition, the fact that the knowledge score is not measured and the relationship between the knowledge score and the anxiety score is not evaluated should be considered as a limitation. Finally, it is also a limitation that the questionnaire form, which includes measurement tools, is applied online. It is thought that measuring errors can be minimized if questionnaires are applied face to face.Practical implications>The results of the study showed that the training given to the workers in the communication sector contributed positively to the reduction of anxiety levels. It is important to provide training and support to those with high anxiety levels. After providing effective protection for nurses/midwives under pandemic conditions and preventing their uncertainties, they can contribute to the reduction of anxiety levels by providing training to individuals who serve the society. It is recommended to plan health trainings for the anxiety of other sector workers serving the society and to focus on these groups. Thus, the effective protection of individuals and their service quality will increase and their anxiety may decrease.Originality/value>Informative support from nurses/midwives can make it easier to control anxiety arising from the COVID-19 pandemic. The results are important in order to draw attention to the anxiety of other sector workers serving the society and the importance of informative roles of nurses. In order to reduce the anxiety levels of workers in different sectors, it is recommended to conduct more supportive training activities and to draw attention to the workers serving the society.

Accuracy versus reliability-based modelling approaches for medical decision making.

Forecasting in the medical domain is critical to the quality of decisions made by physicians, patients, and health planners. Modeling is one of the most important components of decision support systems, which are frequently used to simulate and analyze under-studied systems in order to make more appropriate decisions in medical science. In the medical modeling literature, various approaches with varying structures and characteristics have been proposed to cover a wide range of application categories and domains. Regardless of the differences between modeling approaches, all of them aim to maximize the accuracy or reliability of the results in order to achieve the most generalizable model and, as a result, a higher level of profitability decisions. Despite the theoretical significance and practical impact of reliability on generalizability, particularly in high-risk decisions and applications, a significant number of models in the fields of medical forecasting, classification, and time series prediction have been developed to maximize accuracy in mind. In other words, given the volatility of medical variables, it is also necessary to have stable and reliable forecasts in order to make sound decisions. The quality of medical decisions resulting from accuracy and reliability-based intelligent and statistical modeling approaches is compared and evaluated in this paper in order to determine the relative importance of accuracy and reliability on the quality of made decisions in decision support systems. For this purpose, 33 different case studies from the UCI in three categories of supervised modeling, namely causal forecasting, time series prediction, and classification, were considered. These cases were chosen from various domains, such as disease diagnosis (obesity, Parkinson's disease, diabetes, hepatitis, stenosis of arteries, orthopedic disease, autism) and cancer (lung, breast, cervical), experiments, therapy (immunotherapy, cryotherapy), fertility prediction, and predicting the number of patients in the emergency room and ICU. According to empirical findings, the reliability-based strategy outperformed the accuracy-based strategy in causal forecasting cases by 2.26%, classification cases by 13.49%, and time series prediction cases by 3.08%. Furthermore, compared to similar accuracy-based models, the reliability-based models can generate a 6.28% improvement. As a result, they can be considered an appropriate alternative to traditional accuracy-based models for medical decision support systems modeling purposes.

Predicting clinical outcomes among hospitalized COVID-19 patients using both local and published models.

BACKGROUND: Many models are published which predict outcomes in hospitalized COVID-19 patients. The generalizability of many is unknown. We evaluated the performance of selected models from the literature and our own models to predict outcomes in patients at our institution. METHODS: We searched the literature for models predicting outcomes in inpatients with COVID-19. We produced models of mortality or criticality (mortality or ICU admission) in a development cohort. We tested external models which provided sufficient information and our models using a test cohort of our most recent patients. The performance of models was compared using the area under the receiver operator curve (AUC). RESULTS: Our literature review yielded 41 papers. Of those, 8 were found to have sufficient documentation and concordance with features available in our cohort to implement in our test cohort. All models were from Chinese patients. One model predicted criticality and seven mortality. Tested against the test cohort, internal models had an AUC of 0.84 (0.74-0.94) for mortality and 0.83 (0.76-0.90) for criticality. The best external model had an AUC of 0.89 (0.82-0.96) using three variables, another an AUC of 0.84 (0.78-0.91) using ten variables. AUC's ranged from 0.68 to 0.89. On average, models tested were unable to produce predictions in 27% of patients due to missing lab data. CONCLUSION: Despite differences in pandemic timeline, race, and socio-cultural healthcare context some models derived in China performed well. For healthcare organizations considering implementation of an external model, concordance between the features used in the model and features available in their own patients may be important. Analysis of both local and external models should be done to help decide on what prediction method is used to provide clinical decision support to clinicians treating COVID-19 patients as well as what lab tests should be included in order sets.

Deep Learning-Based COVID-19 Pneumonia Classification Using Chest CT Images: Model Generalizability.

Since the outbreak of the COVID-19 pandemic, worldwide research efforts have focused on using artificial intelligence (AI) technologies on various medical data of COVID-19-positive patients in order to identify or classify various aspects of the disease, with promising reported results. However, concerns have been raised over their generalizability, given the heterogeneous factors in training datasets. This study aims to examine the severity of this problem by evaluating deep learning (DL) classification models trained to identify COVID-19-positive patients on 3D computed tomography (CT) datasets from different countries. We collected one dataset at UT Southwestern (UTSW) and three external datasets from different countries: CC-CCII Dataset (China), COVID-CTset (Iran), and MosMedData (Russia). We divided the data into two classes: COVID-19-positive and COVID-19-negative patients. We trained nine identical DL-based classification models by using combinations of datasets with a 72% train, 8% validation, and 20% test data split. The models trained on a single dataset achieved accuracy/area under the receiver operating characteristic curve (AUC) values of 0.87/0.826 (UTSW), 0.97/0.988 (CC-CCCI), and 0.86/0.873 (COVID-CTset) when evaluated on their own dataset. The models trained on multiple datasets and evaluated on a test set from one of the datasets used for training performed better. However, the performance dropped close to an AUC of 0.5 (random guess) for all models when evaluated on a different dataset outside of its training datasets. Including MosMedData, which only contained positive labels, into the training datasets did not necessarily help the performance of other datasets. Multiple factors likely contributed to these results, such as patient demographics and differences in image acquisition or reconstruction, causing a data shift among different study cohorts.

External validity of phase III trials on vaccines against SARS-CoV-2 to a middle-aged and elderly Western European population.

Initial results from various phase-III trials on vaccines against SARS-CoV-2 are promising. For proper translation of these results to clinical guidelines, it is essential to determine how well the general population is reflected in the study populations of these trials. This study was conducted among 7162 participants (age-range: 51-106 years; 58% women) from the Rotterdam Study. We quantified the proportion of participants that would be eligible for the nine ongoing phase-III trials. We further quantified the eligibility among participants at high risk to develop severe COVID-19. Since many trials were not explicit in their exclusion criterion with respect to 'acute' or 'unstable preexisting' diseases, we performed two analyses. First, we included all participants irrespective of this criterion. Second, we excluded persons with acute or 'unstable preexisting' diseases. 97% of 7162 participants was eligible for any trial with eligibility for separate trials ranging between 11-97%. For high-risk individuals the corresponding numbers were 96% for any trial with separate trials ranging from 5-96%. Importantly, considering persons ineligible due to 'acute' or 'unstable pre-existing' disease drastically dropped the eligibilities for all trials below 43% for the total population and below 36% for high-risk individuals. The eligibility for ongoing vaccine trials against SARS-CoV-2 can reduce by half depending on interpretation and application of a single unspecified exclusion criterion. This exclusion criterion in our study would especially affect the elderly and those with pre-existing morbidities. These findings thus indicate the difficulty as well as importance of developing clinical recommendations for vaccination and applying these to the appropriate target populations. This becomes especially paramount considering the fact that many countries worldwide have initiated their vaccination programs by first targeting the elderly and most vulnerable persons.